No logs visible at Panorama-Log collector with ES cluster health in blank
23909
Created On 12/07/22 10:10 AM - Last Modified 12/12/22 08:37 AM
Symptom
- Logs from a log collector are not visible in Panorama.
- ES cluster health is red or blank when running the command >show log-collector-es-cluster health.
- The process "reportd" consumes more memory than expected, generating a memory leak.
Environment
- PAN-OS is 10.1.X below 10.1.8.
- Panorama.
- Log collector in logger mode or mixed mode.
Cause
- There is a well-known issue. PAN-189270, where the "reportd" process is increasing its memory consumption continuously, generating a memory leak.
- ES process tries to be restarted but when re-start again, not enough memory is available, failing, for that reason, the command "show log-collector-es-cluster health" result is blank.
Resolution
- Confirm the memory consumption growth by reviewing the mp-monitor.log.4 and mp-monitor.log:
>less mp-log mp-monitor.log.4
reportd 9347 6 39 15430744 13388360 S
>less mp-log mp-monitor.log
reportd 9347 0 39 16886844 14264012 S
*The above output confirms both, the memory consumption grows and the value is higher than expected. This expected value can vary, depending on your Panorama setup.
- To fix the issue:
- Upgrade to PAN-OS 10.1.8 for a permanent fix.
- Since the upgrade is not instantly possible, run the next commands:
>debug software restart process reportd
>debug elasticsearch es-restart option all
*The ES can take hours to be fully synced, you should wait some time after introducing the commands.